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An Annual Evaluation Report 
Based on School System Improvement Efforts* 

Robert A. Gordon and Linda J. Dickson 

South Metro Psychoeducational Program 



The South Metro Psychoeducational Program, serving students 
with severe emotional/behavioral disorders (birth-21 years) in 
a regional, community based approach, implemented a school 
improvement plan with all other school system instructional 
components. Results indicated statistically significant improve- 
ments in the percent of students moving to fewer segments of 
specialized services, in the percent of students ages 6+ years 
who showed gains in reading for the school year, and in the 
percent of students ages 16+ years who demonstrated an adult 
functional reading level at the 6th year grade level. Both ele- 
mentary and adolescent students demonstrated statistically sig- 
nificant gains on the Whlker-McConnell Scale of Social 
Competence and School Adjustment compared to a comparison 
group in another school system. Discussion of results and rec- 
ommendations for future school improvement efforts are pro- 
vided. 

The need to improve schools has been a clarion call from multiple audiences. Particular 
emphases on improving American education increased dramatically after Sputnik was 
launched. Formal federal actions to encourage and stimulate school improvement have 
included the National Defense Education Act (1965) to increase student achievement in 
science and mathematics; the National Assessment of Education Programs (NAEP) (1968) 
to document student achievement and give direction for continued school improvement 
efforts followed by the National Assessment Governing Board (1987) (as reported by Jones, 
1996); the Nation at Risk (1983) which focused on the imperatives for educational reform; 
the national goals of America 2000: Education of America Act (1990) which provided 
direction in preparing for school, school completion, student achievement and citizenship, 
and parental participation, and public school management by private businesses (GAO, 
1996). Similar state efforts have included “blueprints” for improvement by the Year 2000, 
the Kentucky Education Reform Act (Taylor, 1996), shared decision making (Weiss & 
Cambone, 1994), charter schools (Garcia & Garcia, 1996), and the Quality Basic Education 
Act in Georgia (1985 as amended). While there have been some who have predicted doom 
for our public education system, there are also those who have concluded that academic 
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achievement is not declining in our schools and that there have been improvements in 
student achievement (Berliner & Biddle, 1996; Slavin, 1996). 

Regardless of the reader’s point of view, each school in each state across the country 
can improve its educating students to the maximum of their abilities. Some have suggested 
that in order to be effective, schools must define the problem and collaborate with others in 
the creation of effective policies rather than the creation of rhetoric (Garcia & Garcia, 1996; 
Good, 1996; Taylor, 1996); that true reform must be created at the local level involving 
input from all of the stakeholders including parents, parent groups, and school councils 
(Johnson-Howard, 1991; Shannon, 1994); and that adopting proven practices involving 
teachers using effective instructional methods is the only way that student achievement can 
be maximized (Slavin, 1996; Wang, Haertel, & Walberg, 1993). Regardless of the type of 
school improvement implemented, the particular school improvement strategies carry with 
them an expectation for enhanced educational productivity (Smylie, Lazarus, & 
Brownlee-Conyers, 1996). 

Consistent with state blueprints for progress (Georgia Blueprint for the Year 2000, 
1995), local school systems and local schools have developed individualized designs for 
school improvement. The Clayton County Public Schools’ Blueprint for Progress (1996) 
reflected the development of seven related strategic objectives and strategies (collaboration, 
communication, curriculum and instruction, facilities and operations, fiscal management,* 
staff development, and human resources) based on their mission and belief statements. Ten 
indicators of progress were specified based on this blueprint including improvement on 
curriculum-based assessments, state assessments, and nationally standardized tests. Each 
school in the Clayton County Public Schools, including those with programs which serve 
students with disabilities, developed its individualized School Improvement Plan containing 
goals, objectives, activities/strategies, person(s) responsible, time lines, benchmarks, and 
evaluation-all consistent with the parameters of the Blueprint for Progress (1996). 

The South Metro Psychoeducational Program (Program) serves students with severe 
emotional disturbance /behavioral disorders (SE/BD) ages birth-21 years. Consistent with the 
recommendation from the National Association of School Boards’ report Winners All 
(1992) to include all programs/schools as one, the Program (for which the Clayton County 
Schools is the fiscal agent) developed a school improvement plan for school year 
1996-1997. Based on the seven strategic objectives and strategies, the Program School 
Improvement Plan (SIP) focused on three areas-presence and participation, academic and 
functional literacy, and personal and social adjustment. 
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Purposes/Hypotheses 



This study had three purposes: 1) to document students’ educational status for all the 
dependent variables contained in the School Improvement Plan in FY 1996-1997 (see Figure 
1); 2) to compare these results to those in FY 1995-1996 to determine educational change; 
and 3) to compare the students’ behavioral results on the Walker McConnell Scale to a 
comparison group of students from another psychoeducational program in Georgia to 
provide an external comparison consistent with the Plan. The School Improvement Plan 
focused on changes in dependent variables specified in a framework of domain, behavior, 
dependent variable, and measurement criteria (see Figure 1). 

The research hypothesis related to the latter two purposes for each dependent variable 
was that there would be statistically significant improvement (relative increase, relative 
decrease) comparing the 1996-1997 student data to the 1996-1995 student data (Phase I) and 
that there would be a statistically significant positive difference between the Program student 
pretest/posttest data as contrasted to the comparison group student pretest/posttest data from 
the other psychoeducational program on the Walker-McConnell Scale for both elementary 
and adolescent students (Phase II). The generalizations for the study were limited to SE/BD 
students served in the Program. 



Methods 



Design 



For Phase I, the research/program evaluation design was a posttest only design 
(Campbell & Stanley, 1963) for each of the two fiscal years with multiple dependent 
measures for the educational objectives and outcomes. The baseline data were collected 
from the 1995-1996 school year and the current data were collected in the 1996-1997 school 
year. The sample of students was stratified by chronological age of the students as of 
September 1, 1996. Descriptive and inferential statistics were used to analyze these data. 
For Phase II, the design was a pretest/posttest control (comparison) group design (Campbell 
& Stanley, 1963) using the Walker-McConnell Scale (Elementary and Adolescent) subtest 
and total scores as the dependent variables and the other psychoeducational program 
students as the comparison/control group. 
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Figure 1 : Overview of Independent Variables and Measurement Criteria 
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Sample 



The sample was comprised of students who are educationally diagnosed with SE/BD or 
autism consistent with the requirements of the Individuals with Disabilities Education Act 
(as amended in 1997) and state and federal rules and regulations. For Phase I, the students 
in the sampled attended the South Metro Psychoeducational Program for either one or both 
of the years of study. Because of the characteristics of the students, the sample sizes vary 
for different dependent variables. For Phase II, the students in the sample had complete 
descriptive and pre-test/post-test data on the Walker-McConnell Scale (Elementary or 
Adolescent) during FY 1996-1997. The students in the comparison/ control group met 
similar criteria. Table 1 contains the descriptive data for the Phase I sample. Table 2 
contains the descriptive data for the Phase II samples. 



School Improvement Efforts 



There were three means of focusing resources to implement the school improvement 
plan. The first was the use of a new educational model-the Boys Town Educational Model. 
The second was shared governance including the refinement of a mission statement, the 
development of a leadership team with specification of responsibilities, and the refocusing of 
efforts to impact academic achievement as a central theme. The third was a refocusing of 
instructional efforts for student academic achievement. 

New Educational Model. During 1995-1996 the South Metro Psychoeducational 
Program, previously using a combination of more traditional educational treatment 
approaches, transitioned into the use of the Boys Town Educational Model (BTEM) (Dowd, 
et al., 1993). The BTEM was the exclusive model used with students in FY 1996-1997. The 
BTEM is firmly grounded in the principles of applied behavior analysis and social learning 
theory. It provides a structure and a plan for working with students in the classroom 
settings. It focuses on teaching in an effective, positive manner so that students learn how to 
interact appropriately with others and subsequently achieve academic success. 

The flexibility that is needed to teach SE/BD students considering their problems and 
different situations is contained in the four BTEM components-Social Skills Curriculum, 
Teaching Interaction, Motivation Systems, and Administrative Intervention. Each teacher 
has his/her own teaching style and the BTEM can be used with each teacher. The combined 
use of these components provides a technology to educate students-from those in the most 
restrictive setting to those in the less restrictive setting. 
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Table 1 

Demographic Descriptors of South Metro Psychoeducational Program 
for 1995-1996 (95-96) and 1996-1997 (96-97) 



Grade 


Caucasian 
95-96 96-97 


Male 

African-Am 
95-96 96-97 


Other 

95-96 96-97 


Caucasian 
95-96 96-97 


Female 
African-Am 
95-96 96-97 


Other 

95-96 96-97 


Pre-K 


4 


2 


14 


4 


1 


0 


1 


3 


3 


0 


0 


0 


K 


0 


1 


7 


4 


0 


0 


0 


0 


1 


2 


0 


0 


1st 


3 


2 


8 


6 


0 


0 


0 


0 


1 


0 


0 


0 


2nd 


3 


1 


11 


6 


0 


0 


0 


0 


2 


3 


0 


0 


3rd 


5 


2 


13 


11 


1 


0 


2 


0 


2 


2 


0 


0 


4th 


3 


2 


17 


9 


0 


0 


2 


1 


2 


3 


0 


0 


5th 


4 


6 


19 


9 


1 


0 


1 


0 


1 


0 


0 


0 


6th 


9 


6 


22 


18 


2 


2 


2 


2 


1 


3 


0 


0 


7th 


12 


13 


14 


26 


3 


3 


2 


3 


3 


4 


0 


0 


8th 


9 


16 


14 


22 


3 


0 


2 


8 


7 


5 


0 


0 


9th 


16 


23 


16 


22 


1 


8 


3 


3 


7 


7 


0 


0 


10th 


11 


13 


11 


5 


1 


1 


1 


4 


5 


3 


0 


0 


11th 


6 


9 


2 


11 


0 


2 


0 


2 


3 


4 


0 


1 


12th 


2 


7 


5 


4 


0 


0 


1 


1 


0 


4 


0 


0 


Totals 


87 


103 


173 


167 


13 


16 


17 


25 


38 


40 


0 


1 



The foundations of the BTEM include identifying prosocial behaviors, using proven 
instructional strategies to teach those behaviors, providing incentives for learning and using 
positive behaviors, and implementing consistent and nonpunitive discipline procedures with 
each student. These foundations provide for the creation of a school and classroom 
atmosphere in which each student has the opportunity to reach his/her full potential. The 
BTEM provides ways to empower students to change and control their own behavior in their 
own way, which ultimately leads to less disruption and more learning in the classroom. 

The following sixteen skills are the core of the BTEM behavioral/social curriculum: 
Following instructions, accepting criticism or a consequence, accepting “no” for an answer, 
disagreeing appropriately, giving criticism, greeting others, getting the teacher’s attention, 
making a request, resisting peer pressure, making an apology, engaging in a conversation, 
giving compliments, accepting compliments, volunteering, reporting other youth’s behavior, 
and introducing yourself. The use of the BTEM addressed two of the three areas for school 
improvement-Domain A and Domain C (see Figure 1) 
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Table 2 



Descriptive Information— South Metro Students and Control Group— 1996-1997 







Elementary 


Adolescent 






South Metro 


Control 


South Metro 


Control 


Variables 




n=26 


n=21 


n=40 


n=35 


Sex Male 




88% 


95% 


78% 


97% 


Female 




12% 


5% 


22% 


3% 


Race African-American 


54% 


29% 


63% 


23% 


Caucasian 




46% 


71% 


35% 


77% 


Other 




0% 


0% 


2% 


0% 


Age (In Months) 


Range 


107-163 


108-150 


130-210 


120-210 


Mean 


128.12 


129.81 


170.85 


167.14 




S.D. 


12.42 


12.93 


20.97 


17.09 


GPN Months in 


Range 


0-36 


1-77 


0-70 


0-72 


Treatment 


Mean 


9.50 


28.38 


15.70 


19.26 




S.D. 


10.61 


24.51 


17.33 


16.63 


Months in 


Range 


4-8 


N/A 


4-10 


N/A 


Boys Town Model 


Mean 


5.27 


N/A 


5.70 


N/A 


S.D. 


1.04 


N/A 


1.64 


N/A 


# Days Attended 


Range 


4-115 


32-124 


18-113 


27-121 


Classes in 1996-1997 


Mean 


99.46 


100.45 


86.20 


95.80 




S.D. 


23.11 


24.63 


18.10 


22.99 



The implementation of the BTEM demanded that the instructional staff become more 
specific with their feedback to students. BTEM teaches that staff should lower tolerances for 
inappropriate behaviors so that generalization of appropriate behaviors occurs more easily 
based on societal norms. With the combination of low tolerances and specific, observable 
descriptions of behavior seen and of behaviors expected, communication in the classrooms 
with the students and among the staff is improved. 

Another key aspect of the BTEM is a token economy which is based on the principles 
of reinforcement and response cost. Behavioral principles are reviewed with the staff so that 
they may link the BTEM teaching strategies with research based procedures for teaching 
new skills. Concepts of shaping, modeling, extinction, reinforcement, and punishment are 
revised with the staff during intensive and continuing training throughout the year. The 
power of the motivation system is built on its concept as a level system-moving from the 
acquisition stage of the daily Points Level through the fluency -building stage of the Progress 
Level to the proficiency stage of the Merit Level. 
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Teaching Interactions are taught so that staff will respond to each behavior as necessary 
for each individual student. Preventive teaching is done for new skills and to reinforce skills 
which are not used frequently. Effective Praise is a four step interaction which reinforces 
the correct use of a skill. A Complete Teaching Interaction is a corrective procedure which 
points out the inappropriate response and also teaches the correct response for that situation. 
Ongoing Teaching Interaction provides the staff with a set of four tools to use when a 
student is noncompliant and does not respond to initial correction. Finally, an Office 
Referral uses an interventionist to help the student regain instructional control. 

The BTEM is as powerful as the reinforcers used consistently for each student. 
Considerable time is spent on developing the reinforcer menus and varying them to keep the 
students motivated. The effectiveness of the BTEM is continually enhanced through the 
Consultation component which is implemented throughout the year with support from 
certified Boys Town Trainers for each staff member. 

Shared Governance. Another aspect of the school improvement effort has been the 
development and refinement of a shared governance program incorporating representatives 
from the instructional, clinical, secretarial, custodial, and cafeteria staff who comprised a 
nine-person Leadership Team. This team has a Constitution and functions to advise the 
Administrative staff on building issues and curricular issues. This approach has resulted in 
improved communication and focus for instruction throughout the Program and has 
enhanced the problem solving capabilities of the Team. 

Refocused Instructional Efforts. The academic instructional treatment addressed 
Domain B and incorporated several different approaches. Instruction in reading focused on 
the Char-L Intensive Phonics (1993) for all SE/BD students to supplement instruction in the 
areas of reading, spelling, and writing which may occur as part of the regular curriculum. 
The Quality Core Curriculum (1995) objectives, as well as texts provided by Clayton 
County Board of Education, the Fulton County Board of Education, and the Atlanta Public 
Schools were used for academic instruction. These efforts refocused instruction at the 
individual classroom. 

Summary. Combining these three school improvement efforts facilitated an emphasis 
on classroom improvement activities in terms of classroom management, metacognitive and 
cognitive approaches, student teacher social interactions, social and behavioral emphases, 
motivation and affective dimensions, the quantity of instruction, and developing a school 
culture within a positive classroom climate emphasizing classroom instruction-consistent 
with the knowledge base for school learning developed by Wang, Haertel, and Walberg 
(1993). 




13 



9 



GPN Research Report 
Volume 7, 1998 



Analyses 



For Phase I, the descriptive analyses consisted of the proportion (and number) of 
students meeting the measurement criterion of the total number of students in that subgroup. 
A test of proportionality (Ferguson, 1956) was used to test the significance of the 
differences between the proportions for FY 1995-1996 and FY 1996-1997. For Phase II, the 
descriptive analyses consisted of the ranges, means, and standard deviations for both the 
Elementary and Adolescent versions of the Walker-McConnell Scale. Two analyses were 
conducted to determine the significance of the differences between the various groups for 
each of the relative subtests and total scores. The dependent t-test was used to determine the 
significance of the differences between the pretest and the posttest using the student as 
his/her own control. The second was an analysis of covariance using the pretest as a 
covariate and comparing the posttest from the comparison group and BTEM students. 



Results 



Phase I 



The summary data analyses for Phase I are contained in Table 3. The data are 
presented by domain by dependent variable by FY of data with the corresponding z-score 
for the test of proportionality. For Domain A (Presence and Participates) , there were 
significant differences in the proportions for the two fiscal years for percent of students who 
moved to fewer segments in Participates--one for the B-14 age group and one for the 154- 
age group-z-score= 1 .73 (significant beyond the .10 level) and z = 3.38 (significant beyond 
the .01 level). Thus there was a significant increase (ranging from double to almost triple) 
in the proportion of students moved to receive fewer segments of special education across 
the two fiscal years. For the the percent of 12th grade students who received a completion 
document (Completes), no test could be performed because in both bases there was no 
variance to analyze; however, all of the students who were in the 12th grade in both years 
received completion documents--a standard which cannot be improved upon-and thus a 
ceiling effect was evidenced. For the second variable in Completes School, there was a 
reduction in the proportion of students who dropped out but the difference was not 
statistically significant. 

For Domain B (Academic and Functional Literacy) there were significant differences 
for both the variables concerning Demonstrates Competence in Reading Skills. There was a 
statistically significant difference in the percent of students ages 6-1- who showed a gain in 
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Table 3 



Summary of Descriptive and Inferential Data Analyses—Phase I 



Domain 


Dependent Variable 


Baseline 
FY 1995-1996 
% 

(Sample Size) 


Current 
FY 1996-1997 
% 

(Sample Size) 


Z-Score 


A: Presence 
and Participation 


Participates 

1 . % of students ages B-14 
who have moved to fewer 
segments 


4.93% 

(10/203) 


9.62% 

(15/156) 


1.73* 




2. % of students ages 15+ who 
have moved to fewer segments 


7.97% 

(11/138) 


23.53% 

(24/102) 


3.38*** 




Completes School 










1 . % of 12th grade students 
who receive a completion 
document from Psychoeduca- 
tional Program or regular school 


100% 

(8/8) 


100% 

(12/12) 


NA 




2. % of students ages 16+ 
who drop out 


19.57% 

(27/138) 


14.71% 

(15/102) 


.98 


B: Academic and 
Functional Literacy 


Demonstrates Competence in Reading Skills 

1. % of students ages 6+ who 76.19% 
show gain in reading for (112/147) 

school year 


91.30% 

(126/138) 


3.44*** 




2. % of students ages 16+ 
who demonstrate adult func- 
tional reading level at 6th 
grade level 


18.26% 

(21/115) 


37.93% 

(33/87) 


3.13*** 


C: Personal and 
Social Adjustment 


1 . % of students requiring 
psychiatric hospitalization other 
than for diagnostic evaluations 
during year 


0% 

(0/358) 


0% 

(0/268) 


NA 




2. # of students placed in 
residential setting during year 
by IEP 


0 

(0/358) 


0 

(0/268) 


NA 



Test of Proportionality, z(.Ol) = 2.58***, z (.05) = 1.96**, z(.lO) =1.65*, NA = No analysis 
because of zero variance (i.e., proportion for FY 1995-1996 = proportion for FY 1996-1997) 
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reading for the school year. This proportion increased to over 90% in FY 1996-1997 and 
was significant beyond the .01 level. For the percent of students ages 16+ who 
demonstrated adult functional reading levels at the 6th grade level, the proportion increased 
by double and was also statistically significant beyond the .01 level. Thus students made 
significant progress in reading achievement across the program. 

For Domain C (Personal and Social Adjustment) the two variables studied in Phase I 
could not be statistically analyzed because there was no variance. For the first variable, 
there were no students who required psychiatric hospitalization other than for diagnostic 
evaluations during either fiscal year. For the second variable, there were no students who 
were placed in residential settings per the IEP for either fiscal year. While there was no 
variance to analyze, the standard of no students requiring residential services cannot be 
improved upon. 

Phase II 

The summary data for Phase II are contained in Table 4. The dependent t-tests using 
each student as his/her own control revealed six statistically significant differences for the 
nine variables. For the Elementary group (n=26), there were statistically significant 
differences for all subtest scores and for the total score: Teacher-Preferred (t = 3.49, p.01), 
Peer Preferred (t=6.16, p.001), School Adjustment (t=3.30, p.01), and Total Score 
(t=3.61, p.01). Thus, the Elementary group of 26 students made statistically significant 
gains in Personal and Social Adjustment as measured by the Walker-McConnell Scale over 
a four month period using the Boys Town Educational Model. For the Adolescent group, 
gains were achieved in the four subtests and the total score. The Peer Relations and the 
Total Score results indicated statistically significant growth for this group of 40 Adolescent 
students using the Boys Town Educational Model. 

The descriptive and inferential data for the comparison/control group from another 
psychoeducational program are also contained in Table 4 for informational purposes. A 
review of these results reveals an inconsistent pattern of growth over the testing period and 
indicates no statistically significant differences for either the Elementary or Adolescent 
groups using the Walker-McConnell Scale. 

Two analyses of covariance, using the pretest as the covariate, the posttest as the 
dependent variable, and the comparison/control group from the other psychoeducational 
program vs. the group served by the BTEM at the Ash Street Program as the main effect, 
were conducted— one for the Elementary Group and one for the Adolescent Group (See 
Table 5). For the Elementary group, there was a significant difference between the 
comparison/control and the BTEM groups (F(l,45) = 4.93, p.03). For the Adolescent 
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Table 4 

Summary of Descriptive and Inferential Analyses— Phase II 
Walker-McConnell Scale of Social Competence and School Adjustment Pretest vs. 



Posttest Comparisons for FY 1996-1997 
Dependent t-tests Using Percentile Rank 







Pretest 




Posttest 


t-value 




Mean 


SD 


Mean 


SD 




Elementary Version 
(n~26) 

Teacher-Preferred 


39.00 


11.12 


48.92 


11.97 


3.49** 


Peer Preferred 


45.62 


15.77 


58.15 


13.02 


6.16*** 


School Adjustment 


28.27 


7.68 


34.23 


8.37 


3.30** 


Total Score 


116.58 


29.67 


140.38 


31.02 


3.61** 


Adolescent Version 
( n=40 ) 

Self Control 


29.65 


9.09 


32.50 


11.70 


1.70 


Peer Relations 


47.72 


11.72 


52.90 


9.98 


2.88** 


School Adjustment 


40.43 


10.08 


42.43 


13.28 


1.15 


Empathy 


16.90 


6.47 


17.43 


7.46 


.41 


Total Score 


142.18 


31.70 


155.00 


38.06 


2.68* 


Control Group from another Psychoeducational Program 








Elementary Version 
(n-21) 

Teacher-Preferred 


43.19 


17.21 


42.04 


12.69 


0.43 


Peer Preferred 


51.24 


19.10 


53.19 


13.83 


0.65 


School Adjustment 


29.04 


10.72 


30.05 


9.15 


0.61 


Total Score 


122.95 


41.61 


126.43 


34.46 


0.51 


Adolescent Version 
(n — 67) 

Self Control 


30.29 


10.40 


28.68 


11.21 


0.96 


Peer Relations 


47.54 


11.22 


43.94 


15.22 


1.71 


School Adjustment 


41.45 


10.94 


39.69 


13.73 


0.87 


Empathy 


15.03 


5.13 


14.63 


6.25 


0.44 


Total Score 


141.91 


34.78 


135.3 


44.26 


1.01 



*p.05; **p.01; ***p.001 
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Table 5 

Analysis of Covariance for Walker-McConnell Scale-Elementary 
Covariate = Pretest 



Source of Variation 


Sum of Squares 


DF 


Mean Square 


F 


Sig 


Covariate(Pretest) 


13680.723 


1 


13680.72 


18.144 


.000 


Main Effects 


3715.32 


1 


3715.32 


4.93 


.032 


BTEM 


3715.32 


1 


3715.32 


4.93 


.032 


Explained 


17396.04 


2 


8698.02 


11.54 


.000 


Residual 


33930.44 


45 


754.01 






Total 


51326.48 


47 


1092.05 






Raw Regression Coefficient for Pretest: .487 














Adolescent 






Source of Variation 


Sum of Squares 


DF 


Mean Square 


F 


Sig 


Covariate(Pretest) 


45409.54 


1 


45409.54 


39.85 


.000 


Main Effects(BTEM) 


7933.20 


1 


7933.20 


6.69 


.01 


Explained 


53342.74 


2 


26671.37 


23.40 


.000 


Residual 


83190.46 


45 


1139.60 






Total 


136533.20 


47 


1820.44 







Raw Regression Coefficient for Pretest: .487 
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Group, there was a statistically significant difference between the comparison/control and 
the BTEM groups (F(l,73) = 7933.20; p.01). Thus, the results indicated that all 
students-both Elementary and Adolescent-made significantly greater progress being served 
in the BTEM versus the comparison/control group using more traditional models. 



The combination of school improvement efforts-BTEM, shared governance, and 
focused instruction-resulted in statistically and practically significant results in multiple 
areas. For Phase I analyses focusing on benchmark student achievement variables, 
significant results were found for seven of the eight variables (four were statistically 
significant, the other three were maintained either at appropriate ceiling or zero levels). 
Changes for all eight of these variables were in the appropriate direction; changes for all 
eight variables were incremental in nature, that is they increased/decreased in partial 
amounts rather than an “all or none’’ change sequence. This pattern of changes in the 
appropriate directions suggests that a foundation has been established which not only 
resulted in significant increases in student achievement but also will foster continuing 
changes in these benchmark variables. Further, it may be possible to generalize these results 
to other critical areas, e.g., student achievement in mathematics might be considered as 
another target area. 

Concerning the Phase II analyses, the results suggest both statistical and practical 
significance for most of the areas measured for both elementary and adolescent students. 
The results indicate that more significant growth may be anticipated from elementary 
students (who made gains across multiple subtests and totals) than adolescent students 
perhaps because younger students have behavior patterns which are less established than 
those of adolescents. Nevertheless, these datasuggest that the BTEM is an effective model 
for improving student achievement of both elementary and adolescent students in the 
personal and social adjustment domain. 

These student achievement results, based on the combination of school improvement 
efforts, are consistent with the research of Wang, Haertel, and Walberg (1993) concerning 
the implementation of the most effective school improvement activities, e.g., instructional 
strategies, to impact on student achievement. The use of the shared governance approach as 
an integral part of school improvement efforts suggests that it will continue to facilitate 
school improvement efforts. Including this Program serving students with disabilities as an 
integral part of the Clayton County School System School Improvement Planning 
demonstrates that special education programs can be incorporated as vibrant contributors to 
the student achievement in all areas. These results provide a foundation to consider refming 
this process and the foci for continued student achievement in 1997-1998. 



Discussion 
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Summary 



The South Metro Psychoeducational Program, which serves students with severe 
emotional/behavioral disorders from birth through age 21 years in a regional, community 
based approach, developed a school improvement plan based on its fiscal agent school 
system’s Blueprint for Progress consistent with actions by all other schools in the system. 
The plan included specific indicators in three domains-presence and participates, academic 
and functional literacy, and personal and social adjustment. Integral parts of the school 
improvement plan was the incorporation of the Boys Town Educational Model (BTEM) and 
leadership emphases targeting the increase of student achievement. Comparisons were made 
on the specific indicators for the 1995-1996 and 1996-1997 school years for elementary and 
adolescent students served in one or both of these years. Statistical analyses included the test 
of proportionality, dependent t- tests, and analyses of covariance. Results indicated 
statistically significant improvements in the percents of students B-14 years and 15+ years 
moving to few segments of specialized services,, in the percent of students ages 6+ years 
who showed gains in reading for the school year, and in the percent of students ages 16 + 
years who demonstrated an adult functional reading level at the 6th year grade level. 
Additionally, the percent of 12th grade students who received a completion document was 
maintained at 100% for both years and the percent of students requiring psychiatric 
hospitalization for any reason was maintained at 0% for both years. Also, both elementary 
and adolescent students demonstrated statistically significant gains on the Walker-McConnell 
Scale of Social Competence and School Adjustment in six of the nine subtests and total test 
scores. Statistical comparisons to students not served with the BTEM (control group) 
revealed that students served in this Program with the BTEM made statistically significant 
gains as compared to the control group. Discussion of results and recommendations for 
future school improvement efforts provide for improving efforts for 1997-1998.. 



* Appreciation is expressed to Dr. Ron Thompson, Director of Evaluation Research, 
and Penny Ruma, Research Assistant, at Boys Town USA for conducting the descriptive 
and inferential analyses depicted in Tables 2, 4, and 5 as a part of the GPN-Boys Town 
Research Consortium. 



Notes 
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Staff Training Strategies for Implementing the Boys 
Town Education Model in Georgia Psych oeducational 
Network Programs 

Linda J. Dickson 

South Metro Psychoeducational Program 

This study investigated the strategies used training 154 staff 
(both certificated and nonce rtiflcated) in the implementation of 
the Boys Town Educat ional Model in 51 classes in four Georgia 
Psychoeducational Network Programs in 1996-1997, the pilot 
year. Descriptive analyses for twelve questions, including the 
identification of practices that were effective and ineffective, 
along with discussions and implications for continued training 
efforts in 1997-1998. 

Since the Georgia Psychoeducational Network (GPN) began with the prototype 
program funded in 1970, various educational treatment models have been used within the 
individual programs to provide a structure for the delivery of instruction and for addressing 
the behaviors of students with severe emotional/behavioral disorders. The GPN consists of 
24 state funded, community-based, non-residential, locally operated units which serve 
multi-county geographic areas linking services with local school systems, area community 
mental health programs, and regional mental health facilities (Stansberry, 1989). The GPN 
provides psychoeducational services to severely emotionally/behavioral ly disordered 
students and their families in every school system in the state (Swan, Wood, & Jordan, 
1991). 

Dr. Mary M. Wood introduced the Developmental Therapy Model at the Rutland 
Center in Athens, Georgia in 1969. (Swan et al. y 1991). The Developmental Therapy 
Model, which emphasized the areas of behavior, socialization, communication, and 
academics as the components of a curriculum guide for teachers of students with severe 
emotional problems, was taught in small classes for a portion of the day and the students 
were placed for the remainder of the day in regular classrooms, day care facilities, or other 
structured programs where they could be around more typical role models and try out their 
new skill steps. 

As Georgia and federal education laws changed, so did special education programs 
within each school district in Georgia. The special education programs in local districts 
delivered services to a moderately disabled population and demanded that the services for 
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students with severe emotional/behavioral disorders change in order to serve even more 
severely disabled students. If the psychoeducational programs were to remain an extension 
of the school district’s special education program in behavior disorders and the last step 
before residential placement in the cascade of services, the programs had to be able to deal 
effectively with behaviors with which other professionals’ interventions had not been 
successful. The psychoeducational programs had to consider self-contained placements as an 
option on the continuum of services, thus requiring modification(s) of the Developmental 
Therapy Model and/or changes in other models. 

Stansberry’s (1989) study “Training Needs of Fully Certified BD Teachers in the 
Georgia Psychoeducational Network” identified training needs in the areas of social 
skills/affective education, strategies for dealing with job related stress, better understanding 
of and involvement with emotionally disturbed students, and classroom structure. The 
Training Needs Questionnaire was developed and used with staff from the 24 
psychoeducational programs to assess staff development needs. School age and adolescent 
teachers reported common training needs while preschool teachers identified needs which 
were somewhat different than the other groups. Curricula for social skill development and 
vocational programming were consistently seen as high priority needs identified by teachers. 
Other training needs were found to be the areas of motivating/rewarding students and 
individual counseling sessions. 

During 1995, four Directors of Psychoeducational Programs were introduced to the 
Boys Town Educational Model (BTEM). The Boys Town Education Model (Dowd, Tobias, 
Connolly, Criste, & Nelson, 1993) is rooted in the principles of applied behavioral analysis 
and social learning theory. It provides a structure and a plan for working with students in a 
classroom setting, even those students who are disruptive and at-risk. BTEM’s focus is on 
teaching in an effective, positive manner so that students learn how to interact appropriately 
with others, and subsequently to find academic success. In the four components-the Social 
Skills Curriculum, Teaching Interactions, Motivation System, and Administrative 
Intervention-is the flexibility that is needed in different situations when dealing with 
students who have social/behavioral problems. The model uses best teaching practices and 
focuses on instruction and reinforcement of the skills for individual success. The foundation 
of BTEM involves identifying prosocial behaviors, using proven instructional strategies to 
teach those behaviors, providing incentives for learning and using positive behaviors, and 
implementing consistent and nonpunitive discipline procedures. It provides the ability to 
empower students Ur change and control their behaviors which ultimately leads to less 
disruption and more learning in the classroom. There are 16 critical social skills which are 
task analyzed and taught step-by-step to each student. Other social skills are also taught as 
determined by student needs. These educational strategies have been used at Boys Town in 
Nebraska since 1979 and have been proven to enhance student success. 
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Whelan and Simpson (1996) of the University of Kansas have suggested that personnel 
who work with children and youth with severe emotional/behavioral challenges must to be 
skilled in implementing research-based practices to assure an appropriate education for these 
students. Administrative and program staff of South Metro Psychoeducational Program, 
Cobb-Douglas Psychoeducational Program, Alpine Psychoeducational Program, and 
Southwest Georgia Psychoeducational Program believe that accountability is critical for 
quality programming and that the Boys Town Education Model provides a re search- based 
model tested and used over the past 15 years. Why reinvent the wheel to develop a model 
for the psychoeducational programs when an effective model is working with a population 
of students identified with emotional/behavior disorders in Nebraska? 



Staff Development 



In September, 1995, 10 educational leaders from Georgia attended a five day 
Specialized Classroom Management Workshop conducted by the National Education 
Training Center staff of Father Flanagan’s Boys Home in Nebraska. Following the training, 
these staff members committed to completing the requirements to become certified trainers 
in the Boys Town Education Model so that the model could be effectively replicated in these 
four GPN programs. The four Directors of the programs involved received funding under 
three Georgia Special Education Incentive Grants to secure funding for training in 
Specialized Classroom Management, Consultation, Administrative Intervention, Common 
Sense Parenting, and Family Preservation. All of these training areas are specific programs 
used at Boys Town. Each training session was conducted in Georgia with the Training Team 
from Boys Town and included intensive interaction and training for the ten professionals 
working toward Training of Trainers certification. Training was held in Georgia in 
December, January, February, March, April, May, June, and August, 1996. Following the 
August Trainers’ Institute, nine trainers were certified to conduct training in the Boys Town 
Education Model for staff in their programs and school districts. The tenth trainer was hired 
by Cobb-Douglas Psychoeducational Program from the Boys Town staff. Collaboration with 
Boys Town in Nebraska is ongoing. 



Method 



To evaluate the impact of me Boys Town Educational Model training and 
implementation from the trainers’ perspective, questionnaires were sent to each trainer 
asking for their perceptions and experiences with BTEM in their programs after using the 
model for at least 90 school days (1/2 an academic year). All questionnaires (10/10 = 100%) 
were returned. The overwhelming response from the four programs represented was a 
positive one with trainers unanimously stating that they would commit the time for training 
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again if necessary due to the perceived effectiveness of the model. Each item on the 
questionnaire was analyzed and the results presented below. 



Results 



The responses to each item are presented individually. A brief discussion follows the 
presentation of the results. 



Item ffl 



This item focused on the number of staff members who had been trained in Specialized 
Classroom Management and were still working with the model at the time of the responses 
to the questionnaire. A total of 154 personnel (see Table 1) had been trained at the four 
Programs ranging from 20 to 52 across the four programs. The large majority of those 
trained were teachers followed by paraprofessionals, administrators, and social workers. 



Table 1 

Summary of Responses to Item #1 

Item # 1: Please list the number of staff members by job category who have been 
trained in Specialized Classroom Management and who are working with your 





Alpine 


Cobb/Douglas 


South Metro 


SWGa 


Totals 


Teachers 


14 


23 


20 


14 


71 


Paraprofessionals 


0 


18 


3 


15 


36 


Administrators 


4 


4 


9 


2 


19 


Social Workers 


1 


4 


3 


5 


13 


Psychologists 


1 


1 


0 


1 


3 


Other* 


0 


2 


5 


6 


13 


Totals 


20 


52 


40 


42 


154 



*Speech/Language Therapists, Art Therapists, Interventionists, Middle School Personnel, Behavioral 
Intervention Specialist/Behavior Specialist, Custodian, Secretary, Director 
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Item n 



The second item focused on the number of classrooms operating using the required 
components of the Boys Town Educational Model-Social Skills Curriculum, Teaching 
Interactions, Motivation Systems, and Administrative Interventions. Respondents indicated 
that a total of 51 classrooms were using the required model components-37 school-age 
(B-14 years) and 14 high school (15-18 years) classes. The number of classes across the 
programs ranged from a high of 17 (South Metro) to a low of 6 (Alpine) (see Table 2). 

Table 2 

Summary of Responses to Item #2 

Item #2: Please list the number of classrooms operating in your Program which are 
using the following components of the Boys Town Educational Model: Social Skills 
Curriculum, Teaching Interactions, Motivation Systems, and Administrative 



Intervention. 





Alpine 


Cobb/Douglas South Metro SWGa 


Totals 


School-Age 


3 


11 


12 


11 


37 


(B-14 Years) 












High School 


3 


4 


5 


2 


14 


(15-18+ Years) 












Totals 


6 


15 


17 


13 


51 



Item #3 



Item #3 focused on the approach to training of staff used by each of the Programs. 
Cobb/Douglas used two approaches-training all staff at the Cobb site and training certified 
personnel and paraprofessionals at the Douglas site. Southwest Georgia also focused on 
training all staff; the Alpine and South Metro Programs focused on training certified staff 
and then others, phasing in the training when possible for other staff. Certified staff include 
those members with professional teaching credentials from the Teacher Certification 
Division of the Georgia Professional Standards Commission including teachers, social 
workers, psychologists, speech/language therapists, and art therapists. 
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Item #4 



The respondents indicated that the four Programs had used a variety of techniques to 
support the staff during the first year of the implementation of the model (see Table 3). 
Among the highest rated techniques were extensive implementation planning, leadership 
teams, partners in education, consultation procedures, and skill enhancement through 
retraining, study groups, weekly discussion groups, and cooperative brain storming. 



Table 3 

Summary of Responses to Item #4 

Item # 4 : Please rate three things you have done to support staff during this first year 
of implementation and rate the effectiveness. (Rating of 1-3 with 1 being high) 


Rating 


Support Techniques 


1 


Extensive Implementation Planning (SWGa) 

Leadership Team (Staff representatives advising administrators regarding 
curricular concerns) (South Metro) 

Establishing Partners in Education (Businesses to help fund the Motivation 
System Rewards) (South Metro) 


2-3 


Skill Enhancement: Retraining (Cobb/Douglas) 

Study Groups (South Metro) 

Weekly Discussion Groups (Alpine) 
Cooperative Brainstorming (SWGA) 


1-2-3 


Consultation Procedures (All Four Programs) 


Other: 


Offered continued staff development 

Working with/modeling for staff during stressful times 

One-on-one discussions 

Individual meetings 

Visual social skills planning 

Reviewed de-escalation techniques 

Hired a person for Interventions 



Item #5 



Concerning feedback which was received from staff who had been trained and were 
using the model with students in a classroom setting, the following comments were listed: 
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Model is difficult to implement and requires more work than systems previously used; 
teachers are pleased with results in the classroom of core group of students; positive overall 
responses-some talk about stress levels; most staff are positive-two teachers do not think it 
works with their students; students are making progress-however, staff must play too many 
roles; more training sessions are needed so all staff in building are trained; Teaching 
Interactions take time-staff must balance teaching behavior and teaching academics; staff 
opinions vary from enthusiastic and supportive to non-compliant; four out of six teams like 
the model and desire proficiency-two elementary classes had successful systems in place 
before and do not see the same benefit with the change; feedback ranges from showing 
excitement to pessimism-model is difficult to implement and has increased the paperwork; 
feedback has been very positive-program is difficult to implement; and staff members 
express that office referral consequences are not severe enough. The array of feedback 
covers the range from enthusiastic to pessimistic. 



Table 4 

Summary of Responses to Item #6 

Item #6: Describe three significant impacts that the BTEM has had on your program. 

# of Trainers Significant Impacts of BTEM on Program 

4 Change in focus from managing inappropriate behavior to teaching skills for 

appropriate behaviors 

4 Improved consistency throughout Program (data collection and teaching) 

4 Increased professional expectations of staff and administrators- Accountability 

3 Decrease in coercive intervention techniques 

2 More mainstreaming occurring 

2 Positive comments from parents and regular communication with parents 

1 -Improved relationships through increased feedback 

-Speaking the same language (staff and students) 

-More success is seen in academics 
-Student progress is related to specific skills 
-The model rejuvenated a teaching team 
-The model organizes feedback to teachers 

-It provides clear guidelines for what constitutes an Office Referral 

-Students are success-oriented 

-Provides a structure and system that students prefer 

-Need for increased support staff 

-Extinction bursts during first six months of treatment 
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Item #6 



The responses to the request to describe three significant impacts that the Boys Town 
Model has had on their Program emphasized a change in focus from managing inappropriate 
behaviors to teaching skills for appropriate behaviors and improved consistency throughout 
the Program (data collection and teaching) along with increased professional expectations of 
staff and administrators. Table 4 summarizes all the responses to this item. 



The following were listed by trainers describing the most significant difficulties 
experienced in implementing the BTEM: Staff taking “short cuts” and undermining the 

model (n=5 trainers), not enough staff for Administrative Interventions (n=5 trainers), lack 
of time for training (n=4 trainers), lack of commitment from some staff (n = 3 trainers), no 
time for consultation (n=3 trainers), and maintaining morale due to increased 
responsibilities (n=2 trainers). Other responses included the expression of dealing with a 
core group of middle school age students without resorting to Time-Out and the extinction 
bursts which resulted in an increase of out-of-control behaviors. Trainers also indicated that 
giving support to staff without fostering dependence was a new challenge and the time it 
takes for consultation to be effective and staff to be supported is overwhelming. Money is 
limited to operate a point store and a different use of space is needed for program 
implementation. One trainer commented that changing mindsets of staff, parents, and 
students which say that negative points are bad has been a difficulty the program has had to 
address. 



For this item, trainers responded with suggestions for redesigning the BTEM training 
approach with their Program. Four responses were shared by multiple trainers: Have all 
staff trained in Specialized Classroom Management prior to implementation-consider use of 
Pilot Teams (n=4 trainers); would not do differently (n=2 trainers); more Ongoing 
Interaction training (n=2 trainers); more role-play time (n=2 trainers). Other responses 
from trainers included the following: Increased emphasis on problem-solving, increased 
focus on integration with speciality services, continued staff education, more 
general ization-IEP relationship to goals and objectives and to levels system, emphasis on 
anger control strategies, and focus on what to do with students with repeated Office 
Referrals without improvement. 



Item til 



Item ti8 
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Item #9 



This item asked trainers if after having used the BTEM in their programs, they would 
commit the time if the training had to be conducted again. All respondents (10/10 = 100%) 
answered “yes”. 



This item was designed to seek information concerning changes in staff, students, or 
parents which the trainers would attributed to the implementation of the BTEM. Three 
responses accounted for the majority of responses--More reports of skill generalization to 
home, school, and church than formerly heard (n=4 trainers), staff are more descriptive 
when discussing behaviors than prior to training (staff speak same professional language) 
(n=4 trainers), and student progress is traced to specific skills (n = 3 trainers). Other 
responses included: Increased awareness of non-coercive techniques to de-escalate students, 
increased involvement of parents in teaching/reinforcing basic skills due to Common Sense 
Parenting, teachers spend less time behind desks, saved a teacher from burn-out, consistent 
student expectations, good student/staff/parent relationships, positive attitudes, and high 
stress levels in support staff working as Interventionists. 



This item sought information on how trainers were using data generated by the BTEM 
Point Sheets. Four uses accounted for the majority of responses: Measure success on IEP 
goals (n = 6 trainers), consultation (n=6 trainers), evaluation of teachers (n = 3 trainers), and 
use point sheets to look at target areas (n=2 trainers). Other responses included: Treatment 
planning, look for trends, positive/negative interaction ratio, look at skill mastery, feedback 
to parents, and evaluate the effectiveness of Office Referral process. 



This item concerning a projection of how many staff trainers thought would leave at the 
end of the year because of the implementation of the BTEM. Responses ranged from 0-2 
(Alpine) to 1-5 (Cobb/Douglas) with the median being between 1 and 2 staff members. 



Item #10 



Item #11 



Item #12 
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Item #13 



This item requested information on whether staff had already left the Program because 
of the implementation of BTEM. Responses ranged from a low of 0 (Alpine, SW Ga.) to 
1-2 for Cobb/Douglas. 



Summary/Discussion 



The Boys Town Trainers in the Georgia Psychoeducational Network have trained 154 
staff members of the four pilot programs during the first half of the academic year 1996-97. 
Future training programs are being planned in each of the four programs to continue staff 
training to reach the goal of fully training program staff in the model. Additionally, other 
psychoeducational programs have had staff trained in the model. 

There are 51 classrooms using the Boys Town Education Model in the four pilot 
programs of the Georgia Psychoeducational Network. Training and implementation is 
occurring in satellite classes, in special education feeder classes in the associated districts, 
and in other psychoeducational programs. 

The Cobb Psychoeducational Program and the Southwest Georgia Psychoeducational 
Program have trained all staff. The remaining programs in the pilot are continuing to train 
staff during phase in of the model. Comments from trainers reflecting staff feedback, 
difficulties in implementation, and redesigning the training approach included references to 
the desire to have all staff trained prior to implementation. 

Feedback from the staff noted the difficulty of using the model due to its specificity and 
increased teaching time and paperwork. Each program addressed support of staff in a 
variety of ways, yet all the programs overwhelmingly rated “Consultation Procedures” as 
an important technique for staff support. 

The trainers indicated that the adoption of the Boys Town Education Model has 
improved consistency throughout the programs in the area of teaching and in the area of 
data collection. The same language is spoken by staff, students, and parents in addressing 
social skills and learning. The crux of the model is the change from simply managing 
behaviors to the teaching of behavioral skills and the specific, identified steps of those skills. 
With the teaching and data collection comes accountability for the teaching and acquisition 
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of skills. As students acquire the skills and move through the level systems at a pace with 
which they feel comfortable, more mainstreaming occurs based on the student improvement. 

The comprehensiveness of the model addresses intervention techniques which are 
non-coercive and consequently increases the professional expectations of all staff involved 
with the model at any job level. The impact on each of the four pilot programs has been 
positive in spite of difficulties perceived and the model has reinforced accountability and 
success-oriented programming in the psycho educational programs involved. 

Perceived difficulties during the first year of a major change in programming include 
staff taking “short cuts” with the interventions and reverting to management techniques 
which they have used in the past. Increased responsibilities and shortage of staff for new 
roles must be accommodated for as changes are made. Time for support and consultation 
seems also never to be enough as the model is implemented. Some trainers would adapt the 
schedule/ scope of the training in order to address these difficulties if they had the year to do 
over again but other trainers would use the existing design of to train and implement the 
model in their programs. 

Few staff are projected to leave the pilot programs due to the adoption of the Boys 
Town Education Model. If the trainers’ are accurate in their predictions that no more than 
15 staff will leave the pilot programs due to the implementation of Boys Town Education 
Model out of the more than 154 staff trained, the percentage of those leaving would be 
10%. According to Hamm (1996), the mean attrition rate for teachers in the 
psychoeducational network for the period 1986-1992 was 17.86% per year and the mean 
attrition rate for paraprofessionals was 18.31% per year, for the same time period. 
Therefore, the staff leaving the pilot programs is not seen as significantly variant from 
previous years. 

Finally, the trainers were asked how they used the data generated by the model. The 
responses show that the measure of student success for IEP goals and objectives were 
obtained from the data collected. Staff support and evaluation was done through data 
collection. Additionally, programs reviewed trends, skill mastery, and did treatment 
planning through the use of data. 



The training process taught by the Boys Town National Training Team and used by the 
pilot programs in the Georgia Psychoeducational Network has proven to be successful. 
Further study needs include staff attitudes over time, attrition over a three year period, and 
student success as measured by increase in the number of mainstreamed segments. Analysis 
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of Office Referral data and analysis of positive to negative teacher interactions over time 
will be other areas of critical data on which program and staff evaluation and feedback can 
be based. The importance of training will only be as good as the support for staff members 
which is available on an ongoing basis. Future studies should bear this hypothesis out. 

Decisions regarding additional program training and implementation should be based 
on results of student progress. Further studies to document student change in behavior and 
academics and to compare staff satisfaction with the model versus their satisfaction with 
other approaches to educational treatment will provide administrators and leadership teams 
information to use in model evaluation. 
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Implementation of the Boys Town Education Model in 
Four Georgia Psychoeducational Network Programs: 
Initial Impact on Student Social Skills and Adjustment 

Ronald W. Thompson, Penney R. Ruma, Cathy S. Nelson, and 

Andrea H. Criste 

Father Flanagan ’s Boys ’ Home 



This study evaluated the implementation of the Boys Town 
Educational Model (BTEM), a social skills training model, 
with severely emotionally disturbed/behaviorally disordered 
students served in the Georgia Psychoeducational Network. 
The treatment group ( received BTEM) was composed of 189 
students (61 elementary and 128 adolescent); the control group 
( did not received BTEM) was composed of 56 students (21 
elementary and 35 adolescent). The dependent measure was 
the Walker-McConnell Scale of Social Competence and School 
Adjustment. Results indicated that students in the treatment 
group made significant gains in social skills and school 
adjustment while students in the control group did not. 
Discussion of the results are provided. 



Approximately 15 years ago a seminal report (Knitzer, 1982) focused the nation’s 
attention on the general lack of needed programs and services for children and adolescents 
with serious emotional/behavioral disorders (SED). An extensive study of programs an 
policies for SED children eight years later concluded that much still remained to be done to 
design effective programs for these children (Knitzer, Steinberg, & Fleisch, 1990). One of 
the main conclusions of these studies was that, in practice, the priority of many of the 
school curricula for SED children was behavior management with emphasis on maintaining 
classroom silence rather than teaching children appropriate ways of expressing their feelings 
and interacting with other children. One approach which does focus on teaching positive 
behaviors and has demonstrated some promise is social skills instruction (Gresham, 1981). 
Nevertheless, there is limited evidence of whether or not social skill instruction can be 
successfully implemented over time in special education programs, and if it can produce 
long term benefits for these children. 

In this study the Boys town Educational Model (BTEM) was implemented in four 
regional programs in the Georgia Psychoeducational Network (GPN). The initial research 
question was to evaluate whether or not students experienced significant improvements in 
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social skills. Long-term research questions referred to whether or not students improved in 
their general adjustment in this special education setting, and if they could be successfully 
mainstreamed into general education programs. 



Method 



Design 



Students enrolled in four regional, community -based GPN programs between July and 
March of the 96-97 school year served as the treatment group for this study. They were all 
enrolled in classrooms staffed by teachers trained in the BTEM. Students enrolled in 
satellite centers of one of these programs served as a control group. None of the control 
group children were enrolled in classrooms staffed by teachers trained in the BTEM. 
Students were divided into elementary (K-6) and adolescent (7-12) subgroups for the study 
based on the research of Hill Walker and his colleagues indicating that somewhat different 
social skills are observed at these two age levels (Walker & McConnell, 1995). Similar data 
were collected on all students. 



Participants 



There were 189 students in the treatment group. Sixty-one were elementary students 
and 128 were adolescents. These groups had a majority of males, and they were almost 
equally split between African American and Caucasian students. They had been enrolled in 
the GPN program from 0-146 months at the beginning of the study, and they attended 
anywhere from 2 to 160 days during the year. Descriptive statistics for the treatment group 
are provided in Table 1. 



There were 56 students in the control group. Twenty-one were elementary students and 
35 were adolescents. They were almost all males, and the majority were Caucasian. They 
had been enrolled in the GPN program from 0-77 months at the beginning of the study, and 
attended from 27 to 124 days during the year. Descriptive statistics for the control group are 
provided in Table 2. 



Treatment and control students were similar in most respects except, that treatment 
students were enrolled in self-contained special education programs and control students 
were enrolled in satellite programs. There were, however, two other obvious differences 
between the treatment and control groups according to the descriptive data. First, the control 
group had a much higher percentage of Caucasian students. Second, control elementary 
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Table 1 



Descriptive Information— Overall Treatment Group 



Variable 


Mean 


Elementary 

(n=61) 

SD Range 


Mean 


Adolescents 
(n= 1 28) 

SD Range 


Chronological Age 


127.59 


13.80 


95-163 


172.02 


21.94 


128-257 


in Months 
Months Enrolled 


10.87 


11.54 


0-44 


18.08 


19.63 


0-146 


in GPN 

Months in BTEM 


5.59 


0.96 


3-8 


6.43 


1.91 


4-14 


Treatment 

Days Attending School 


103.10 


20.18 


4-160 


92.58 


24.14 


2-159 


Gender 

Male 

Female 

Race 

African American 

Caucasian 

Other 




85% 

15% 

53% 

46% 

1% 






81% 

19% 

40% 

59% 

1% 











Table 2 










Descriptive Information- 


■Control Group 






Variable 




Elementary 




Adolescents 






(n— 2 1 ) 






(n=35) 






Mean 


SD 


Range 


Mean 


SD 


Range 


Chronological Age 


129.81 


12.93 


108-150 


167.14 


17.09 


120-210 


in Months 














Months Enrolled 


28.38 


24.51 


1-77 


19.26 


16.63 


0-72 


in GPN 














Months Enrolled in 


NA 


NA 


NA 


NA 


NA 


NA 


BTEM Treatment 














Days in School During 


100.45 


24.63 


32-124 


95.80 


22.99 


27-121 


1996-1997 














Gender 














Male 




95% 






97% 




Female 




5% 






3% 




Race 














African American 




29% 






23% 




Caucasian 




71% 






77% 




Other 




0% 






0% 
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students had been enrolled in GPN programs almost three times as long as treatment 
elementary students. 



Program 



The GPN consists of 24 community-based day programs that are part of the public 
school continuum of services for children with severe emotional and behavior disorders and 
autism. Comprehensive special education services are provide din self-contained specialized 
schools and satellite programs located in general education settings. The goal is to maintain 
these students in the community as a cost-effective alternative to residential placement and 
to return these students to less restrictive settings in their local communities. Each class is 
staffed by a lead teacher and a paraprofessional. Other specialists, including school 
psychologists and social workers also provide support services. These support services 
include assessment of student and family needs, psychological and psychiatric consultation, 
parent training, and counseling. In addition, there is an emphasis on coordination with other 
agencies which provide services to the students and their families. Of the four programs 
involved in the study, two were located in a major metropolitan area, and the other two 
were in smaller communities. 



Boys Town staff provided training to administrators and instructional staff in these four 
GPN programs during the 95-96 school year, and all had begun implementation of the 
BTEM at the start of the study. The BTEM is a systematic school-based intervention which 
includes a Detailed approach to classroom behavior management and social skills training 
(Dowd, Tobias, Connolly, Criste, & Nelson, 1993). The BTEM is an adaptation of the 
Boys Town Family Home program (Coughlin & Shanahan, 1986) and the Teaching Family 
Model of group home treatment (Phillips, Phillips, Fixsen, & Wolf, 1973) to the school 
setting. A longitudinal study of the Boys Town Family Home program indicated long-term 
educational effects for at-risk children and adolescents in residential care (Thompson et al . , 
1995). The BTEM includes a social skills curriculum, a structured approach to classroom 
management focusing on increasing positive behaviors, an office referral intervention 
component and specialized classroom management for the special education setting. BTEM 
evaluation studies have indicated positive effects for general education elementary students 
(Furst et al . , 1995), but the model has not yet been systematically evaluated in special 
education settings. 



Initial training was provide din the form of two workshops, a five-day Specialized 
Classroom Management workshop and a three-day Administrative Intervention workshop 
for building administrators. Following this training Boys Town staff provided consultation 
to classroom teachers to assist with implementation of the model in GPN programs. Finally, 
a team of 10 GPN staff were trained to become trainers and consultants in their own 
programs. 
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Measures 



The dependent measure in this study is the Walker-McConnell Scale of Social 
Competence and School Adjustment (Walker & McConnell, 1995, 1995a). The 

Walker-McConnell was designed to measure social competence. The authors define two 
primary domains in this construct: adaptive behavior and interpersonal social competence. 
They define adaptive behavior as general adjustment, and interpersonal social competence as 
the skills required to maintain successful social relationships. Both these domains include 
teacher-related and peer-related expectations. All items are positive stated. Classroom 
teachers who have observed children in the classroom over time rate their students on each 
item. Each item is a description of a relevant social behavior and it is rated on a five-point 
scale as occurring from never to frequently. The elementary version consists of 43 items 
divided into three subscales: Teacher-Preferred Social Behavior, Peer-Preferred Social 
Behavior, and School Adjustment. These subscales are also combined into A Total Score. 
The adolescent version includes 63 items divided into four subscales: Self-Control, Peer 
Relations, School Adjustment, and Empathy. These subscales are also combined into a Total 
Score. Extensive research has been completed establishing these scales and the reliability 
and validity of the scale itself (Walker & McConnell, 1995, 1995a). 



Results 



An Analysis of Covariance (ANCOVA) using Walker-McConnell total scores was used 
to test for overall BTEM treatment effects. Separate analyses were completed for 
elementary students and adolescents because there are different forms of the 
Walker-McConnell for these two age groups. These analyses are reported in Table 3. The 
source of variation of primary interest is Group. This is a test of the different between post 
test means of the treatment and control groups, accounting for any differences between pre 
test means. There was not a significant treatment effect for elementary students, but there 
was for adolescents (p .05). These results support the conclusion of treatment effects for 
adolescents but not for elementary students. 



Means and standard deviations of subtest and total scores for treatment and control 
groups are provided in Tables 4 and 5. Pairwise comparisons were also made using 
two-tailed dependent t-tests. Examination of these data indicates that both elementary 
students and adolescents in the treatment group made significant gains on the 
Walker-McConnell. On the total score elementary students were, on average, at the 16th 
percentile at pre test and the 22nd percentile at post test. Treatment group adolescents were, 
on average, at the 27th percentile at pre test and the 33rd percentile at post test. Elementary 
treatment students made gains on Teacher-Preferred, Peer-Preferred, and School 
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Table 3 

Analysis of Covariance— Overall Treatment Group 



Elementary 

(n=82) 


Source 


SS 


DF 


MS 


F 


WMTOTPRE 


24133.66 


1 


24133.66 


32.64** 


Group 


681.98 


1 


681.98 


0.92 






Adolescents 








(n=163) 






Source 


SS 


DF 


MS 


F 


WMTOTPRE 


82975.85 


1 


82975.85 


69.56** 


Group 


7568.28 


1 


7568.28 


6.35* 



Note: WMTOTPRE = pretest Walker-Mc Connell Total Score. Groups treatment vs. control. 
*p.05; **p.01 



Table 4 



Walker-McConnell Pre-Post Comparison- Overall Treatment Group 





Pretest Mean 


SD 


Posttest Mean 


SD 


t 


Elementary Version (n= 
Teacher-Preferred 


= 61) 
41.95 


12.12 


47.32 


11.23 


-3.26** 


Peer-Preferred 


51.31 


15.33 


55.31 


13.13 


-2.23* 


School -Adjustment 


30.64 


8.12 


33.20 


7.97 


-2.43* 


Total Score 


125.74 


30.85 


134.46 


30.37 


-2.14* 


Adolescent Version (n= 
Self Control 


- 128) 

31.38 


9.35 


33.17 


11.96 


-1.88 


Peer Relations 


48.76 


13.52 


51.47 


12.58 


-2.20* 


School Adjustment 


43.73 


11.83 


44.63 


13.07 


-0.88 


Empathy 


18.14 


7.25 


17.38 


7.05 


1.03 


Total Score 


147.85 


34.04 


155.80 


39.40 


-2.50* 



*p.05, **p.0l 
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Table 5 

Walker-McConnell Pre-Post Comparison— Control Group 





Pretest Mean 


SD 


Posttest Mean 


SD 


t 


Elementary Version (n 


=21) 










Teacher-Preferred 


43.19 


17.21 


42.04 


12.69 


0.43 


Peer- Preferred 


51.24 


19.20 


53.19 


13.83 


-0.65 


School Adjustment 


29.04 


10.72 


30.05 


9.15 


-0.61 


Total Score 


122.95 


41.61 


126.43 


34.46 


-0.51 


Adolescent Version (n= 


35) 










Self Control 


30.29 


10.40 


28.68 


11.21 


0.96 


Peer Relations 


47.54 


11.22 


43.94 


15.22 


1.71 


School Adjustment 


41.51 


10.94 


39.69 


13.73 


0.87 


Empathy 


15.03 


5.13 


14.63 


6.25 


0.44 


Total Score 


141.91 


34.78 


135.31 


44.26 


1.01 



*p.05, **p.01 



Adjustment (all) subscales, while adolescents made significant gains on only the Peer 
Relations subscale. 

Control group students, however, made no statistically significant changes on the 
Walker-McConnell. On the total score control group elementary students were, on average, 
at the 14th percentile at pre test and the 16th percentile at post test. Adolescents were at the 
24th percentile at pre test and the 19th percentile at post test. 



Discussion 



The initial results reported in this study indicate that students enrolled in GPN 
classrooms using the Boys Town Educational Model made significant gains in social skills 
and school adjustment as measured on a standardized instrument. Students enrolled in 
satellite control classrooms not using the model, however, made no such gains. These 
findings provide preliminary evidence of the effects of the BTEM on students’ social skills 
and school adjustment during the first year of implementation. 

The means and standard deviations seem to contradict the ANCOVA results, however. 
Elementary students made the most consistent gains, and yet treatment effects were found 
only with adolescents. When trying to interpret this type of analysis, it is often helpful to 
graph pre-post means for treatment and control groups. These are provided in Figure 1. 
These graphs clearly indicate that elementary treatment group students improved, but 
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elementary control group students also improved slightly (even though this was not 
statistically significant). When the control group improves even slightly, it makes it difficult 
to get a significant treatment effect in this type of analysis. Perhaps this is why the treatment 
effect for elementary students is not significant. With adolescents, on the other hand, the 
treatment group improved and the control group deteriorated slightly (again not statistically 
significant). This pattern optimizes the opportunity to find a significant treatment effect in 
the ANCOVA. Perhaps this is why the treatment effect was found with adolescents, even 
though the gains across subscales were not as consistent as with elementary students. 

These age effects suggest patterns that may or may not be supported by future 
research. Elementary students in the treatment group made more consistent gains than 
adolescents. Perhaps elementary students are better candidates for social skills instruction 
because poor social skills have not been practiced as long or their effects have not had as 
great a deleterious effect on students’ self competence. In the control group, on the other 
hand, elementary students appeared to improve slightly, and adolescents appeared to 
deteriorate slightly without the treatment program. This may indicate that elementary 
students again are more easily taught social skills, even without a specific social skills 
program. Adolescents, however, may continue to deteriorate in social skills in special 
education placement without specific treatment in that area. 

Several caveats must be considered in the interpretation of these data. First, assessment 
of social skills was limited to one dependent measure. Assessment of social skills has been 
hotly debated in the literature. Methods have included teacher ratings, sociometric 
techniques, and direct observation. The Walker-McConnell is based on teacher ratings. 
Walker and McConnell (1995, 1995a) provide a compelling argument for this approach, but 
clearly assessment of social skills does not have an established methodology at this time. 
Second, treatment group data were collected in four different programs, and the descriptive 
and outcome data looked somewhat different in each program. The results reported 
represent overall averages, and thus are somewhat difficult to generalize to all settings. 
Third, there were some distinct differences in the treatment and control groups which could 
help account for these results. Probably the most systematic bias was that the treatment 
group students were placed in self-contained programs and the control group was limited to 
satellite programs. Fourth, the quality of implementation of the BTEM differed from 
classroom to classroom and program to program. As with any innovation, some teachers 
were more accepting than others and some were able to implement the model and teach 
social skills more quickly than others. Also, GPN consultants/trainers were just learning 
how to help others implement the BTEM in their programs. Because this was the first full 
year of implementation of the BTEM, we hope to find more consistent student 
improvements in future years. 

Despite these cautions, the results provide positive preliminary results with a 
substantial sample of students who have serious emotional/behavioral disorders. They lend 
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support to the hypothesis that social skills training can be implemented with and is beneficial 
to this population. Currently, efforts continue to insure implementation of the BTEM in all 
classrooms in the programs. The team of GPN consultants/trainers from each program have 
received additional training. They meet regularly to provide training to new teachers and 
work with current teachers on implementation issues. Data collection is ongoing. Follow-up 
assessment of social skills and school adjustment is being conducted again during the 97-98 
school year. If students’ social skills are improving, this should also affect other indicators 
of their school adjustment. Therefore, rates and reasons for referrals to the office for 
behavior problems are currently being monitored to identify changes over time. Finally, 
success of mainstreaming students back into general education classrooms is being tracked 
as the long-term goal of this project, specifically, and GPN programs, in general. 
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GPN Report Card— A Status Report 



Carol Pope 

Coastal Georgia Comprehensive Academy 
and 



Chair, Accountability Committee, GPN 

Beginning in 1995, the Accountability Committee of the 
Georgia Psychoeducational Network initiated the development 
of educational outcomes for students with severely emotional 
disturbance/behavioral disorders. Adapting a format and some 
content from the National Center on Educational Outcomes, 
the Accountability Committee developed a statement of 
Educational Objectives and Outcomes (EOO) composed of 
eight data-based items in three domains— Presence and 
Participation, Academic and Functional Literacy, and Personal 
and Social Adjustment. Data from the pilot test in 1995’ 1996 
and from the first full network use in 1996’ 1997 are presented. 
The GPN Report Card grows out of this effort incorporating 
each of the eight items in the EEO and providing additional 
items. Initial data will be gathered on the Report Care in 1998’ 
1999 from all GPN programs. 



The Accountability Committee of the Georgia Psychoeducational Network (GPN) was 
formed in the summer of 1995 at the request of the Coordinator of the GPN in the Division 
for Exceptional Students in the Georgia Department of Education. The purposes of this 
Committee were to address educational outcomes for students with severe emotional 
disturbance/behavioral disorders (SE/BD) and to increase the GPN Program’s accountability 
for outcomes. Based on three years of prior efforts, the Programs achieved a consensus of 
direction on the effort. The members of the Accountability Committee included Dr. Bess 
Allen (Rutland Psychoeducational Program), Mr. David Fallin (Flint Area 
Psychoeducational Program), Ms. Judi Kelley (Cedarwood Psychoeducational Program), 
Ms. Elizabeth LeClair (Middle Georgia Psychoeducational Program), Ms. Susan McKenzie 
(Georgia Department of Education), Ms. Martha Patton (Northwest Psychoeducational 
Program), Ms. Carol Pope (Committee Chair) (Coastal Georgia Comprehensive Academy), 
Dr. Mike Powell (Cobb-Douglas Psychoeducational Program), Dr. Ken Wallin (Coastal 
Academy), and Mr. Homer Wells (Woodall Psychoeducational Program). 

The first meeting was held in September 1995. The Committee utilized the Self Study 



Guide to the Development of Educational Outcomes and indicators from the National Center 
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on Educational Outcomes (NCEO) (Ysseldyke & Thurlow, 1993). The NCEO model uses a 
step-by-step process to create a system of outcomes and indicators needed to develop and 
implement an outcomes-assessment program. There are four major steps: 1. Establish a 
solid foundation for the effort; 2. develop, adopt, or adapt a model; 3. establish a data 
collection and reporting system; and 4. install the system. Seven subsequent meetings were 
conducted throughout 1995, 1996, and 1997 in using this process to develop the Educational 
Outcomes and Objectives (EOO) which is composed of eight data-based items in three 
domains— Presence and Participation, Academic and Functional Literacy, and Personal and 
Social Adjustment. 



EOO Field Test and GPN Data Collection 



The nine directors who participated on the Committee field tested the format and data 
collection system in 1995-1996 resulting in some minor revisions to the initial format and 
procedures. Table 1 contains the EOO with the field test data and data from all 24 Programs 
which were gathered for the 1996-1997 school year. 

Visual inspection of the data in Table 1 show consistency across both years with 
percent differences ranging from 1% to 3% across all eight items. In Domain A-Presence 
and Participation, the data from 1996-1997 for each of the four items were from 1% to 2% 
lower than the field test data. For Domain B-Academic and Functional Literacy, the 
percent differences for 1996-1997 were higher on one item and lower on the other. For 
Domain C, Personal and Social Adjustment, the data from 1996-1997 were lower on one 
item and the same on the second. 



GPN Report Card 



The Accountability Committee has expanded the original eight items from the EOO 
considering data required in other reports cards (e.g., Georgia Department of Education, 
1996; ) and projecting requirements under The Individuals with Disabilities Education Act 
Amendments of 1997. The second draft of the Report Card (see Figure 1) is composed of 
several major components. Student Data is composed of descriptive data on all students 
enrolled. The Educational Objectives and Outcomes includes the original Domain 
A-Participation and completion with items on attendance, students placed in less restrictive 
settings, dropout rate, graduate/post secondary data, and graduates entering post-secondary 
experiences. Domain B-Academic and Functional Literacy includes items on reading, 
functional reading, mathematics, and assessment information (which is not required until the 
1998-1999 school year). And Domain C-Personal and Social Adjustment is composed of 
four items concerning psychiatric hospitalization, placement in residential settings, reduction 
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Table 1 

Educational Objectives and Outcomes (EOO) 
Georgia Psychoeducational Program Network 





1995-1996 

(n=9) 


1996-1997 

(n=24) 


Domain A: Presence and Participation 
Participates 

% of students ages B-14 who move to fewer segments 


18% 


17% 


% of students ages 15+ who move to fewer segments 


23% 


22% 


Completes School 

% of 12th grade students who receive a completion document 
from Psychoeducational Program or regular school 


66% 


64% 


% of students ages 16 + who drop out 


14% 


13% 


Domain B: Academic and Functional Literacy 
Demonstrates competence in reading skills 

% of students 6+ who show gain in reading for school year 


54% 


51% 


% of students ages 16+ who demonstrate adult functional 
reading level at 6th grade level 


43% 


45% 


Domain C: Personal and Social Adjustment 
Copes effective with personal challenges, frustrations, and stressors 
% of students requiring psychiatric hospitalization other 
than for diagnostic evaluations during year 


5% 


4% 


# of students placed in residential setting during year per IEP 


22 


22 



of inappropriate behaviors, and increases in appropriate behaviors. Initial data collection 
efforts using the GPN Report Card are projected for 1998-1999. 
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FIGURE I 

GPN REPORT CARD— DRAFT #2 
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EDUCATIONAL OBJECTIVES/OUTCOMES: 

DOMAIN B: ACADEMIC AND FUNCTIONAL LITERACY* 
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EDUCATIONAL OBJECTIVES/OUTCOMES: 

DOMAIN B: ACADEMIC AND FUNCTIONAL LITERACY (CONTINUED) 
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GUIDELINES FOR A UTHORS 

The GPN RESEARCH REPORT invites manuscripts concerned with any research aspect of program operations for 
severely emotionally disturbed/behaviorally disordered (E/BD) students in Georgia. In order to be considered for 
publication, manuscripts must report or interpret some aspect of data-based scientific finding or practical experience 
that leads to improved understanding of E/BD students or educational programs for these students. 

AGREEMENTS: To be considered for review, a manuscript must meet the following prerequisites: 

1. It is not being considered concurrently by another publisher. 

2. The author assumes responsibility for publication clearance in the event that the manuscript was presented at a 
professional meeting of another organization or was developed by a project funded by an external funding agency. 

3. The manuscript has not been published in substantial part in another journal or published work. 

REQUIREMENTS: In preparing manuscripts for publication, authors must use the following guidelines: 

1- Manuscripts should be well organized and concise. Historical data common to dissertations should be limited. The 
readers of the GPN RESEARCH REPORT use the studies and therefore research findings should be specific, 
concrete, and gleaned from the method and procedures. Implications for the practitioner should be discussed in light of 
the findings. 

2. Manuscript length may vary according to the subject. However, those most acceptable to space available are 
between 8 and 15 double-spaced pages. 

3. A brief abstract of 80-125 words should accompany the manuscript. It should be a clear and succinct statement of 
the treatment, method, and conclusions. 

4. Complete author information should be given on a cover sheet: Full name, title or position, institution, city, state, 
and address with zip code. Grant numbers and funding information should be included when appropriate. The author’s 
name should appear nowhere on the manuscript. 

5. Avoid footnotes. 

6. All material must be double spaced, including quotations and references. Wide margins should be left for editorial 
-work. 

7. References are to follow the style described in the Publication Manual of the American Psychological Association 
(APA, 1200 17th Street, N.W., Washington, D.C. 20036) 

8. Submit a master copy of the manuscript plus four additional copies to William W. Swan/Carvin L. Brown, Editors, 
.GPN RESEARCH REPORT, Department of Educational Leadership, River’s Crossing, the University of Georgia, 

850 College Station Road, Athens, GA 30602. 

9. Authors are responsible for the factual accuracy of their contributions. Manuscripts will be acknowledged upon 
receipt. Following preliminary examination by the editors, the manuscripts will be sent to associate editors/reviewers 
for review. Within approximately three months, the authors) will be notified concerning disposition of the manuscript. 
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